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Abstract 

We prove that a variant of the insertion encoding of Albert, Linton and Ruskuc for 
any class of pattern avoiding permutations is context-senstive. It follows that every 
finitely based class of permutations bijects to a context-sensitive language. 
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1 Introduction 



In this article we consider a connection between the fields of formal language 
theory from computer science, and restricted permutations from combina- 
torics. In particular, we set out to describe the set of all permuations that 
avoid some pattern q in terms of formal languages, in an attempt to enumer- 
ate them. We prove that there is a bijection between the set of permutations 
which avoid q and a context-sensitive language. This language is a variation 
of the "insertion encoding" of Albert, Linton and Ruskuc [1]. 

In formal language theory there is a heirachy of languages of increasing com- 
plexity, starting at regular languages, then context-free, stack, indexed, context- 
sensitive, decidable and recursively enumerable languages, each one strictly 
contained in the previous one. 

In his paper on P-recursiveness and D-finiteness from 91 [2] Gessel states 
that "another possible candidate for P-recursiveness is the problem of count- 
ing permutations with forbidden subsequences" , which Bona strengthens to a 
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conjecture in [3]. Schiitzenberger proved that regular languages have rational 
generating functions [4], and Chomsky and Schiitzenberger proved that unam- 
biguous context-free languages have algebraic generating functions [5] . Beyond 
this, little is known about generating functions for the higher languages. A nat- 
ural question is to ask whether there is a formal language correpsponding to 
D-finite generating functions. We give examples of context-sensitive languages 
whose generating functions are not D-finite. 

In spite of this, the connections between languages and pattern avoidance seem 
to be worth pursuing. An interesting general question is to find the language 
class of lowest complexity (in the heirachy or otherwise) such that the set of 
permatations avoiding any pattern can be described by languages in this class. 
Since the generating function for 1234-avoiding permutations is known to be 
non-algebraic (but D- finite) (see [3] p. 210 and below), we need to look above 
the class of unambiguous context-free for an answer. 

The article is organised as follows. In Section 2 we define the notion of pattern 
avoidance and basis for permutations, and then define context-sensitive gram- 
mars and their machine counterparts. In Section 3 we define the encoding, 
and in Section 4 we prove the main theorem, that the encoding is a context- 
sensitive language. 



2 Prelimiriciries 

A permutation is a finite ordered list of distinct real numbers. Two permuta- 
tions P1P2 ■ ■ - Pn and qiq2 . . .Qn oi the same length are said to be order isomor- 
phic if Pi < pj if an only if < qj for all 1 < i < j < n. 

A permutation pip2 ■ ■ - Pn contains a patten q — qi . . . qk ii k < n and some 
subsequence Pi^ ■ ■ ■ Pi,^ is order isomorphic to q. A permutation avoids a pattern 
if it does not contain it. 

Given a set of patterns qi,q2, ■ ■ ■ the set of all permutations which avoid all of 
the patterns is called a pattern-avoiding class, and the set qi,q2, ■ ■ ■ is called 
its basis. A class is finitely based if there are finitely many q^ defining it. 

Given a basis, one would like to know the number of permutations of each 
length that avoid it. Define Sn{B) to be the number of permutations of length 
n which avoid all the patterns in B. Even for bases of single patterns comput- 
ing this sequence is a difficult problem, and results are known only for quite 
special cases. One can define a formal power series or generating function for 
this sequence. As mentioned in the introduction, the generating function for 
s„(1234) is not algebraic. The asymptotic behaviour of the number of 1234- 
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avoiding permutations was given by Regev [6], and is incompatible with the 
algebraicity of the associated generating function [7]. It is however D-finite. 
An open question is whether for every pattern g, the generating function for 
Sn{q) is D-finite. See [8] for background on D-finiteness. 

Definition 1 (Lineeir-bounded automaton) A hnear-bounded automaton 

is a nondeterministic Turing machine such that for every input w G E*, there 
is some accepting computation in which the tape contains at most \w\ + 1 
symbols. 

Definition 2 (Context-sensitive) A context-sensitive grammar consists of 
three finite sets N, T, P where N is a set of non-terminals which are denoted 
by upper case letters, and includes a distinguished letter S called the start 
symbol, T is a set of terminals which are denoted by lower case letters, and 
P is a set o/ productions of the form a ^ (3 where a G (A^ U T)*N{N U T)* 
and (3 G {N U T)*. The language of a context-sensitive grammar is the set of 
all words in T* that can be obtained by applying some sequence of productions 
starting form S until all letters are terminals. 

For readers familiar with context-free languages, a context-sensitive gram- 
mar is like a context-free grammar where some productions or rules can only 
be applied in the correct "context". Just as context-free languages have a 
machine-theoretic counterpart (pushdown automata), so do context-sensitive 
languages. 

Lemma 3 A language is generated by a context-sensitive grammar if and only 
if it is accepted by a linear-bounded automaton. 

Examples of context-sensitive languages are {a^}Pc^\n > 0}, {{ab"')"'\n > 0} 
and {l^\p is prime} (which are not context-free). Context-sensitive languages 
are closed under union, concatenation, star, intersection and complement [9]. 

There are many other formal languages of interest, that can be described in 
terms of grammars or machines of various types. See [10]. We breifly mention 
stack languages since they have some relevance to this paper. 

A stack automaton consists of a finite state control, a stack and a pointer 
to the stack, such that one can add and delete tokens from the top of the 
stack, as well as move the pointer down into the stack to read-only. There is a 
finite set of input letters, stack tokens, and finitely many transition rules. The 
transisitions are given by a function taking a state of the finite state control 
and an input letter, and outputing a stack instruction and a new state. If the 
input is read keeping the stack operational and the final state is an accept 
state, then we accept, otherwise reject. 
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2.1 Non-D-finite stack and context-sensitive languages 

As noted in the introduction, the fact that regular and unambiguous context- 
free languages have rational and algebraic generating functions respectively, 
leads one to ask whether this connection continues for other classes of lan- 
guages and generating functions. Here we give two examples of context-sensitive 
languages whose generating functions are not D- finite. The question of whether 
there are ambiguous context-free languages with non-D-finite or even non- 
algebraic generating functions, to the authors knowledge at least, is open. 

Consider the language (0*^6*^0*^ ... a*2'=(6*2fc+i) \ ii < i2 < is ■ ■ ■ < i2k+i}- 
While this language is not context-free, is can be recognised using a stack 
automaton as follows. 

Place a $ token on the bottom of the stack. Read o and place a 1 token on 
the stack. Repeat until the first b is read. For each b, move the pointer down 
the stack one token. If an a is read before the pointer is at the bottom of 
the stack, reject. Otherwise if the bottom of the stack is reached, return the 
pointer to the top of the stack, and for each b, place a 1 token on the stack. 
Repeat reversing the roles of a and b, and if the end of the word is reached 
then accept. 

It follows that this language is indexed, and hence context-sensitive. The num- 
ber of strings in this language of length n is equal to the number of ways one 
can partition n as a sum of smaller integers. Since the generating function 
for this set has infinitely many singularities, it cannot be D-finite [7]. This 
language is similar to that in Girgorchuk and Machi [11]. 

Next, consider the language {a" | n is prime }. This language can be recognized 
by a linear-bounded automaton as follows. Start with the word aa . . .a written 
on the tape, with markers as either end. We need to test if the number of as 
has any divisors. Place a mark * on the second a. This gives you a way to 
count "2" on the tape. Suppose the marked a* is i squares from the start. Use 
this to mark every i-th square after a*. If you reach the end of the tape and 
have a multiple of i a's, then reject, else move the mark * up one square to 
count i + 1. (This is effectively the sieve method). 

The fact that this language is context-sensitive is well known and a more 
careful proof can be found in the literature. The number of strings in this 
language of length n is 1 is n is prime and otherwise. The generating function 
for such a set will not satisfy any polynomial recurrence relation, since there 
are arbitrarily long gaps between successive primes, so this language is not 
P-recursive so not D-finite. 

The fact that there are arbitrarily long gaps between primes follows from the 
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prime number theorem, but a cute argument is that the sequence k\ + 2, fc! + 
3, . . . , A;! + A; is a gap of length k — 1 for any k, since each term has a divisor 
of 2, 3, . . . , /c respectively. 



3 The encoding 

We begin this section with a description of the "insertion encoding" [1]. The 
idea of this encoding is to build up a permutation by successively inserting the 
next highest entry in some open slot, starting from a single open slot, until all 
slots are filled. One can insert in one of four different ways: one can insert on 
the left of a slot, on the right, in the middle (creating two slots from one) or 
filling the slot. 

As an example, the instructions "middle,right,left,fill,fiU" or mrlff build the 
permutation 34215 as follows: 

❖ ^❖lo^o21^^3 o 21^^3421o^34215 

In this example, not knowing any better, we performed each insertion on the 
left- most open slot. If we want to use any other slot, we precede the instruction 
by some number of letter t for translate, such that you shift one slot to the 
right for each t preceding each instruction. 

So for the above example, the instructions mrtltff build the permutation 
52134 as follows: 

o^olo^o2 1o^o2 13o^o2 134^52 134 

The original insertion encoding [1] used integer subscripts on the l,r,m,f 
letters to indicate the choice of slot. This means that the alphabet is not finite 
in general. For patterns such as 312 they proved that one can only ever insert in 
the first slot to avoid the pattern, so in this case only the four (unsubscripted) 
letters are required. 

We call the present version the it-encoding which stands for "insert and trans- 
late". 

Definition 4 (it-encoding) Define a codeword be a string of letters I, r, m, f, t, 
subject to the condition that the number oft letters immediately preceding any 
of the other letters x e {/,r, m, /} is at most k where k is the difference be- 
tween the number of m and the number of f letters that appear to the left 
of X, and the codeword ends with an f. Then such a codeword represents a 
permutation p obtained by a sequence of insertions defined by the following 
procedure. 
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Define the next entry to be the number that is to be inserted next into the 
permutation and denote it by i, define the next slot to be number of the slot 
(numbered from left to right) in p into which the insertion will take place, and 
the next letter to be the next letter of the codeword. 

Set p — o, i — 1 and the next slot to be 1. 

Repeat until the last letter has been read: 

Read the next letter of the codeword. 

• // the next letter is t, increment the neoct slot by 1. 

• If the neoct letter is I then replace the neoct slot by io. Increment i by 1 and 
reset the next slot back to 1. 

• // the next letter is r then replace the next slot by oi. Increment i by 1 and 
reset the next slot back to 1. 

• // the next letter is m then replace the neoct slot by oio ( thus increasing the 
number of slots in p by 1). Increment i by 1 and reset the next slot back to 
1. 

• // the next letter is f then replace the next slot by i (thus decreasing the 
number of slots in p by 1). Increment i by 1 and reset the next slot back to 
1. 

Note that the length of the permutation created is equal to the number of non- 
t letters in the code-word. Thus to enumerate permutations in this encoding 
of a given length, we would make use of a two-variable generating function, 
with one variable for non-f letters and the other for the ts. Then we could set 
the second variable to 1. 



4 E{q) is context-sensitive 

Let denote E the set of all legal codewords and E{q) the set of codewords 
representing permutations that avoid the pattern q. In this section we prove 
that E{q) is context-sensitive. The first step is to show that a linear-bounded 
automaton can check that a string of r, m, /, t is a legal codeword. 

Lemma 5 Let w G {/, r, m, /, t}* written on \w\ squares of tape and let n E N. 
Then one can determine whether w is a legal code-word (that is, a word in E) 
for some permutation of length n and return the original string w on the tape, 
using only the \w\ squares in use and at most 0{\w\^) steps. 

Proof. To to this, we need to make sure that the right number of ts appear 
before each non-t letter. Scan the tape (from left to right) until you find an m. 
Move right until either you find another m, or an /. (if you find neither, reject). 
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When you find an m, / pair (with only r,l,t letters in between), overwrite the 
m and / with m* and /*. 

Now scan from m* to /* and mark a single t (that is, replace it by t*) in front 
of every r, I and the final /*, if there are ts in front of them. 

Continue until you cannot find any more unmarked m, f pairs. If there remains 
an m, reject. If there remains an / which is not at the end of the tape, reject, 
(you should still have the f* at the end of the tape). If there remain any 
unmarked t letters, reject. 

If not, then the work has the correct number of ts in front of each non-i, and 
the input ends with an /, so represents a correct permutation. □ 

If X e {/, r, m, /} occurs in a code-word w then the entry of the encoded 
permutation p corresponding to w inserted by x is denoted by Px- 

Lemma 6 If w = WiXW2yw^ is a codeword with x,y E {l,r,m,f} then p^ < 
Py 

Proof. Since entries are inserted in increasing order, and x occurs before y in 
the codeword, then the entry inserted by x is smaller than the entry that is 
inserted at a later time by y. □ 

We will make use of the following lemma. 

Lemma 7 Let w be an it-encoded word written on \w\ squares of tape and 
let x,y E {l,r,m,f} be such that w = WiXW2yw-i,. Then one can determine 
whether p^ occurs before or after Py in the permutation corresponding to w, 
( that is, determine whether the sub-permutation consisting of p^ and Py is 
order isomorphic to 12 or 21 ), and return the original string w on the tape, 
using only the \w\ squares the input is written on and at most 0{\w\'^) steps. 

Proof. By the previous lemma, since x is to the left of Px < Py So x inserts 
the "1" and y inserts the "2". 

The slot in which x is inserted is determined by "counting" the preceding ts, 
and we can do this by marking each preceding t by a *. Suppose there are i 
marked ts, which means x inserts into the {i + l)-th slot. If a; is an / or / then 
there are i slots to the left of x. If x is an r or m then there are (^ + 1) slots 
to the left of x, so star x as well. 

So the number of starred squares will represent the number of open slots to 
the left of X. This number can be changed if an m or / inserts in a slot to the 
left of X. 



7 



Move right towards y. For each m passed, if we determine that m inserts 
into a slot before the position of z, we need to up our count by +1, (that is, 
mark m by a * to signify a shift of slots). To check this, compare the starred 
entries with the ts immediately before this m. At the machine level, this can 
be achieved by scanning to the first t to the left of this m and marking it by 
\ then scan left to the first starred square, which you remark with **, then 
move right to the t to the left of t\ then left and so on until cither you run out 
of starred squares or run out of ts preceding the m. If there are more starred 
entries, then this m indeed occurs before x, so the position of x needs to be 
adjusted (so just star m). If m has more, do nothing. Unmark the t^s and 
remark the ** squares with *. 

If you encounter an / before you get to again if it is determined that this 
/ fills a slot before the position of x then we remove one of the stars from 
the rightmost starred square. We can determine this by comparing starred 
squares with ts preceding this / by the same procedure used in the previous 
paragraph. 

If at any stage there are no starred entries left, then x now hes to the left of 
any open slots, so PxiPy give a 12 sub-permutation. 

So by the time you get to y, the position of x is preserved. Now compare the 
number of ts preceding y with the number of starred entries. If there are j ts 
preceding and i starred squares, then y inserts into the (j + l)-th slot, so 
if i > j + 1 then Px,Py give a 21 sub-permutation, and if i < j + 1 then Px^Py 
give a 12 sub-permutation. Note that the type of y {l,r,m,f) doesn't affect 
this step. 

Each time wc pass an m or / and when we reach y, the machine scans left 
and right (marking with ** and ^) at most \w\ times, and there are no more 
than |w| m,fs between x and y, so this procedure takes Odwp) steps. □ 

Theorem 8 E{q) is accepted by a linear-bounded automaton in polynomial 
time. 

Proof. Let \q\ = k. We will describe the linear-bounded automaton that can 
decide whether to accept or reject a string oil,r,m, f,t letters written on the 
tape, using the tape alphabet l,r,m, /, t, I* ,r* ,m* , f* ,t* . 

First, verify that the input is a legal code- word by enacting the procedure 
given in Lemma 5. Note that the last letter must be an /. This takes at most 
Od^p) steps. 

Next, we systematically run through all ordered A;-tuples of squares, such that 
each square selected has a non-t letter on it. Let x = (xi, 2:2 . . . , x^) be such a 
tuple. 
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We want to test if x is order isomorphic to q. We do this by performing pairwisc 
comparisons using the procedure of Lemma 6. This involves /c comparisons, but 
since g is a fixed permutation, this is a constant. 

Each comparison takes 0(1^^) time and we must repeat this e 0(1^1*^) 
times, so the algorithm described here runs in polynomial time. 

If a sub-permutation that is order isomorphic to q is found, reject, else accept. 
□ 

Since context-sensitive languages are closed under intersection [10] then any 
finitely based class is context-sensitive. We prove this fact explicitly. 

Corollciry 9 The set of permutations which avoid a finite list of forbidden 
subpatterns can be encoded as a language that is accepted by a linear-bounded 
automaton in polynomial time. 

Proof. Suppose qi, . . . ,qk are a finite list of forbidden subpatterns. By Theorem 
8 we can construct linear-bounded automata to accept each of E{qi), such 
that each automaton leaves the original codeword on the tape, and runs in 
polynomial time. Start with a codeword written on the tape, and run the 
automaton to decide E{qi). The automaton accepts or rejects, and leaves the 
codeword on the tape. If it accepts, run the automaton to decide E{q2) and so 
on. If each of the k automata accept, then accept, else reject. The time taken 
is a constant {k) times a polynomial. □ 

4.1 Full encoding is accepted by a stack automaton 

In this section we will show that E is accepted by a stack automaton. 
Proposition 10 E is accepted by a stack automaton. 

Proof. We will describe a deterministic stack automaton that accepts the set 
of all strings of l,r,m,f,t that are codewords. That is, we must check that 
the number of ts immediately preceding any of the other letters is no more 
that the difference between the number of ms and fs preceding it. 

Recall that a stack automaton constists of a finite state control, and single 
stack, and a pointer which points to some entry in the stack. There are three 
main states, qo,<lA,<lF, start, accept and fail respectively. Place the entry root 
on the bottom of the stack. 

First we describe the transition function for the start state qo. Let x be any 
entry of the stack, and let Xr be any entry except the root. 
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• 5{qo, I, x): move the pointer to the top of the stack (that is, go into a state 
Qup and perform single e transitions moving the cursor up one step) and 
continue. 

• S{qo, r, x): move the pointer to the top of the stack and continue. 

• 5{qo, m, x): move the pointer to the top of the stack, add a new entry, and 
continue (so the pointer points to the new entry). 

• ^{Qo, f,Xr): move the pointer to the top of the stack, delete the top entry, 
(so the pointer reverts back one step) and continue. 

• ^{Qoj root): if all the input has been read, move to qa, or if not, move to 
qf, and stop. 

• S{qo, t, Xr): move the pointer down the stack one step. 

• S{qo,t, root): move to qp, and stop. 

The machine reads input letters from left to right, and the height of the stack 
indicates the current sum of number of ms minus /s. When a i is read, the 
machine checks that this sum is not zero (the pointer points above the root). 
For each t read in succession, the pointer moves down the stack, so if the 
number of ts preceding another letter exceeds (m — /) the input is rejected, 
otherwise, if the end of the input is reaches, and the last letter is an / with 
an empty stack, accept. If an / is read on an empty stack before all the input 
has been read, then no slots remain so the string is not a codeword. □ 
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